Deconfounded Visual Grounding

We focus on the confounding bias between language and location in visual grounding pipeline, where we find that is major reasoning bottleneck. For example, process usually a trivial languagelocation association without reasoning, e.g., any query containing sheep to nearly central regions, due most queries about have ground-truth locations at image center. First, frame pipeline into causal graph, which shows causalities among image, query, target underlying confounder. Through know how break bottleneck: deconfounded grounding. Second, tackle challenge confounder unobserved general, propose confounder-agnostic approach called: Referring Expression Deconfounder (RED), remove bias. Third, implement RED as simple attention, can be applied method. On popular benchmarks, improves various state-of-the-art methods by significant margin. Code available at: https://github.com/JianqiangH/Deconfounded_VG.